We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for "scored actors" in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry - sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.

The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems. While there have been promising advances in designing neural networks to harness multimodal data, the enormous success of data augmentation currently remains limited to single-modality tasks like image classification. Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for example, a caption may no longer be a good description of an image after standard augmentations have been applied, such as translation. Moreover, it is challenging to specify reasonable transformations that are not tailored to a particular modality. In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation, an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the relationship between modalities. We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combinations of modalities that have not been previously considered, and (3) achieve state-of-the-art results on a wide range of applications comprised of image, text, and tabular data.

Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn. While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging. In this paper, we first address the obstacles to performing visualizations on ViTs. Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features. We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information. On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers. In addition, we show that ViTs maintain spatial information in all layers except the final layer. In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation. Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.

Deep neural networks are susceptible to shortcut learning, using simple features to achieve low training loss without discovering essential semantic structure. Contrary to prior belief, we show that generative models alone are not sufficient to prevent shortcut learning, despite an incentive to recover a more comprehensive representation of the data than discriminative approaches. However, we observe that shortcuts are preferentially encoded with minimal information, a fact that generative models can exploit to mitigate shortcut learning. In particular, we propose Chroma-VAE, a two-pronged approach where a VAE classifier is initially trained to isolate the shortcut in a small latent subspace, allowing a secondary classifier to be trained on the complementary, shortcut-free latent subspace. In addition to demonstrating the efficacy of Chroma-VAE on benchmark and real-world shortcut learning tasks, our work highlights the potential for manipulating the latent space of generative classifiers to isolate or interpret specific correlations.

机器人的感知目前处于在有效的潜在空间中运行的现代方法与数学建立的经典方法之间的跨道路，并提供了可解释的，可信赖的结果。在本文中，我们引入了卷积的贝叶斯内核推理（Convbki）层，该层在可分离的卷积层中明确执行贝叶斯推断，以同时提高效率，同时保持可靠性。我们将层应用于3D语义映射的任务，在该任务中，我们可以实时学习激光雷达传感器信息的语义几何概率分布。我们根据KITTI数据集的最新语义映射算法评估我们的网络，并通过类似的语义结果证明了延迟的提高。

随着我们远离数据，预测不确定性应该增加，因为各种各样的解释与鲜为人知的信息一致。我们引入了远距离感知的先验（DAP）校准，这是一种纠正训练域之外贝叶斯深度学习模型过度自信的方法。我们将DAPS定义为模型参数的先验分布，该模型参数取决于输入，通过其与训练集的距离度量。DAP校准对后推理方法不可知，可以作为后处理步骤进行。我们证明了其在各种分类和回归问题中对几个基线的有效性，包括旨在测试远离数据的预测分布质量的基准。

低精度算术对神经网络的训练产生了变革性的影响，从而减少了计算，记忆和能量需求。然而，尽管有希望，低精确的算术对高斯流程（GPS）的关注很少，这主要是因为GPS需要在低精确度中不稳定的复杂线性代数例程。我们研究以一半精度训练GP时可能发生的不同故障模式。为了避免这些故障模式，我们提出了一种多方面的方法，该方法涉及具有重新构造，混合精度和预处理的共轭梯度。我们的方法大大提高了低精度在各种设置中的偶联梯度的数值稳定性和实践性能，从而使GPS能够在单个GPU上以10美元的$ 10 $ 10 $ 10 $ 10 $ 10的数据点进行培训，而没有任何稀疏的近似值。

随机微分方程的系统定义了一系列随机波动率模型。尽管这些模型在金融和统计气候学等领域中取得了广泛的成功，但它们通常缺乏在历史数据上条件产生真正的后验分布的能力。为了解决这一基本限制，我们展示了如何将一类随机波动率模型重新塑造为具有专门协方差函数的层次高斯工艺（GP）模型。该GP模型保留了随机波动率模型的电感偏差，同时提供了GP推断给出的后验预测分布。在此框架内，我们从研究良好的域中汲取灵感，以引入新的型号，即Volt和Magpie，这些模型在库存和风速预测中的表现明显超过了基线，并且自然扩展到多任务设置。

对表格数据的深度学习的最新工作表明了深层表格模型的强劲表现，通常会弥合梯度增强的决策树和神经网络之间的差距。除了准确性之外，神经模型的主要优点是它们学习可重复使用的功能，并且在新域中很容易进行微调。该属性通常在计算机视觉和自然语言应用中被利用，在特定于任务的培训数据稀缺时，转移学习是必不可少的。在这项工作中，我们证明上游数据使表格神经网络比广泛使用的GBDT模型具有决定性的优势。我们为表格转移学习提出了一个现实的医学诊断基准，并提出了使用上游数据来通过各种表格神经网络体系结构来提高性能的方法指南。最后，我们为上游和下游特征集不同的情况提出了一种伪特征方法，在现实世界中，特定于表格的问题广泛。我们的代码可在https://github.com/levinroman/tabular-transfer-learning上找到。

尽管低精度优化已被广泛用于加速深度学习，但低精度抽样仍未得到探索。结果，尽管在许多大规模的情况下，采样是不可行的，尽管对神经网络的概括和不确定性估计给予了显着的好处。在本文中，我们提供了低精确的随机梯度Langevin Dynamics（SGLD）的首次研究，这表明其成本可以大大降低而无需牺牲性能，因为它的内在能力处理了系统噪声。我们证明，低精度SGLD与完全精确的梯度累加器的收敛性比在强凸设置中的SGD对应物的量化误差的影响较小。为了进一步启用低精度梯度蓄能器，我们为SGLD开发了一个新的量化功能，该功能保留了每个更新步骤中的差异。我们证明，低精确的SGLD与完整精确的SGLD相当，只有8位在各种深度学习任务上。